A 



STANT COMMISSIONER FOR PATENTS 
PATENT APPLICATION 
ington, D.C. 20231 



DOCKET NO. JP 9-20 0 0- 017 6US 1 
Date : July 27, 2000 



under Rule 1.53(b) 
Transmitted herewith for filing is the Patent Application/ of: 



Inventors : 
For: 



Hulikunta Prahlad Raghunandan 



o 



AN AUDIO SUPPORT SYSTEM FOR CONTROLLING AN E-MAIL SYSTEM IN A 
REMOTE COMPUTER 



Enclosed are: 

X 4 Sheets of Drawings, 

An assignment of the invention to International Business Machines 

Corporation, Armonk, New York 10504. 



5^ 



A certified copy of a 



application . 



Declaration and Power of Attorney, 
Associate Power of Attorney. 

Information Disclosure Statement with form PTO-1449 with references 
attached . 



T]|^ filing fee has been calculated as shown below: 





(Col. 1) 


(Col. 2) 


^OR: 


NO. FILED 


NO. EXTRA 


ffi^SIC FEE 






llllllllllllllllllllllllllllllllllllllll|lllllllllllH 


TOTAL CLAIMS 


18 - 20 = 


0 


INDEP CLAIMS 


3 - 3 = 


0 


1 1 

MULTIPLE DEPENDENT CLAIM PRESENTED 



OTHER THAN A 
SMALL ENTITY 



If the difference in Col. 1 is less than 
zero, enter "0" in Col. 2. 



RATE 


FEE 




$ 690.00 




X $ 18 = 


$ 0.00 


X $ 78 = 


$ 0.00 


+ $ 260= 


$ 0.00 


TOTAL 


$ 690.00 



X Please charge my Deposit Account No. 09-0468 in the amount 
of $ 690.00 . 



X The Commissioner is hereby authorized to charge payment of the 
following fees associated with this communication or credit any 
overpayment to Deposit Account No. 09-0468 . A duplicate copy of 
this sheet is enclosed. 

X Any additional filing fees required under 37 CFR 1.16. 

X Any patent application processing fees under 35 CFR 1.17. 

Respectfully submitted, 

Manny wV^Schecter 
Registration No.: 31,722 
Tel . (914) 945-3252 

Express Label EL600330578US 
Date of Deposit: July 27, 2000 



IBM CORPORATION 

INTELLECTUAL PROPERTY LAW DEPT. 
P.O. BOX 218 

YORKTOWN HEIGHTS, NY 10 598 



Express Label EL600330578US 
Date of Deposit: July 27, 2000 

AN AUDIO SUPPORT SYSTEM FOR CONTROLLING AN E-MAIL 
SYSTEM IN A REMOTE COMPUTER 

Field of the invention: 

5 This invention relates to an audio support system for controlling an e-mail 
system in a remote computer. 

Background of the invention: 

E-mail systems are becoming all pervasive as communication mechanisms, 
especially with the rapid increase in the use of computers and the internet. The 
existing e-mail systems provide a number of features to facilitate 
communication. However, these e-mail systems are not very friendly for people 
with disabilities, as these systems require the user to be in close proximity with 
the computer containing the e-mail system. It is desirable in situations 
involving people with disabilities to have a means for accessing the e-mail 
system remotely and in particular through a voice input mechanism. 

The voice recognition systems are also known. These systems require the user 
to be physically proximal to the computer for operation. 
20 

US Patent 6,061,718 describes an e-mail delivery system for wireless 
communication system. However, this invention is limited to situations 
involving wireless transmissions over large distances using base stations. It is 
therefore targeted only at the mobile user segment and does not address the 
25 requirements of disabled users in a contained environment. Further more, this 
system does not enable the utilization of the full benefits and features available 
from an e-mail system when used directly on a computer. 
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Wireless Application Protocol (WAP) require Internet Service Provider (ISP), 
as an intermediary and are therefore limited. Further, such systems are useful 
only when communication is over a large distances and not in a contained 
environment. 

5 

The objects and summary of the invention: 

The object of the present invention is to overcome the above mentioned 
drawbacks and provide a system for remote access to e-mail facilities in a 
computer system through an audio support mechanism. 

10 

'7; To achieve the said objective, this invention provides a computing system 
consisting of a central processing unit, associated memory, input/output devices 
^ and containing an e-mail system which is characterized in that it includes a 
% remote control device comprising : 
1^1 5 - means to convert speech to text; 

y - means to transmit the said text to the said e-mail system; 

Si - means to receive text from the said e-mail system; 

Q - means to convert the received text into speech; and 

means to select and access the received e-mail in the said e-mail 
20 system. 

The said remote control device (RCD) includes a means to display the received 
email messages. 

25 The said remote control device is customized by the user to speak aloud the 
sender, date, subject, e-mail content and attachment (if required) for incoming 
mail. 
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The said remote control device (RCD) includes means for selecting email 
messages or folder. 

The said remote control device includes means for speaking aloud the summary 
5 of an e-mail folder. 

The said remote control device is linked to the said computer through infra-red 
or ultrasonic or radio frequency waves. 

10 The said e-mail system provides a speech announcement whenever new 
^ messages are received. 

^ The said remote control device comprises: 

^ - a microphone connected to the input of a microphone amplifier, 

^15 - the output of said microphone amplifier is connected to the input 

SJ of an analog to digital converter (ADC). 

- the output of the said ADC is connected to one input of a processor 
p means, 

a second input of said processor means is connected to a Read Only 
20 Memory (ROM) containing text-to-speech conversion and speech- 

to- text conversion software, 

a first input-output of said processor means is connected to 
Random Access Memory (RAM) which contains the speech-to-text 
data, text-to-speech data, and command and status data, 
25 - one output of said processor means is connected to a transmitter for 

transmitting the said data in RAM to a remote computer 
containing an e-mail system, 
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another input of the saicf processor means is connected to a receiver 
which receives the data from said remote computer containing e- 
mail system, 

another input-output of the processor means is connected to a 
5 control panel containing switches and display elements, 

another output of the said processor means is connected to a digital 
to analog converter (DAC) which converts the data received after 
text-to- speech conversion into an analog signal and feeds to the 
input of an audio amplifier which drives a loud speaker to 
10 generate audible speech. 

2J The said processor means is a digital signal processor, or a micro Controller or 
5 an ASIC. 

^ 15 The said ASIC includes the entire circuit except the microphone, speaker, 
%j computer containing e-mail system and control panel, 

5 The said ASIC contains processing means ROM, RAM, transmitter and 
receiver. 

20 

This invention further relates to a computer program product residing on 
computer storage media containing computer readable devices embodied therein 
for causing a computer e-mail system to receive instructions and e-mail content 
from a remote control device and to send status and e-mail content to the said 
25 remote control device. 

The said program product includes the facility for text-to-speech conversion. 
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This invention further includes a computing system consisting of a central 
processing unit, associated memory, input/output devices and containing an e- 
mail system, a method for remotely controlling the said e-mail system using an 
audio support comprising: 

5 

sending e-mail by : 

receiving the speech input from the user, 
converting the speech input into the analog electrical signal, 
amplifying the said electrical signal and converting to digital form, 
10 - converting the said digitized speech to text, 

transmitting the said text to the computer containing e-mail system, 
receiving the transmitted text in the computer containing e-mail 
system, 

transferring the said text to the said e-mail system, 

15 

receiving e-mail by: 

transmission of data from computer containing e-mail system, 

receiving transmitted data at the remote control device, 
converting said received data from text to digitized speech, 
20 - converting the said digitized speech to analog form, 

amplifying the said analog signal and converting to speech for the 
user. 

The email is selected through speech or through control panel. 

25 

The received mail message is displayed on the said remote control device 
(RCD) 
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Brief Description of the Drawings: 

This invention will now be described with reference to the accompanying 
drawings. 

5 Figure 1 shows the computing system having an e-mail system and remote 
control device and speaker-microphone according to this invention. 

Figure 2 shows the remote control device according to this invention. 

10 Figure 3 shows the electronic circuit in the said remote control device. 

Figure 4 shows the block diagram of the software elements used in this 
invention. 

1 5 Detailed Description of the Invention: 

Referring to the drawings. Figure 1 shows the computer containing an e-mail 
system and having a separate remote control device (RCD) with speaker- 
microphone and a corresponding remote control unit (RCU) in the computing 
system. The said remote control unit (RCU) receives the signals transmitted by 
20 the said remote control device (RCD) and passes to the CPU for processing. 
Similarly, it receives the signals from the CPU and transmits to the RCD. 

Figure 2 shows the remote control device (RCD) with its control panel. 

25 Figure 3 shows the electronic circuit of the said remote control device (RCD). 
The output of microphone (1) connects to the input of microphone amplifier (2) 
which supplies an amplified signal to A - D converter (3). The output of the 
said A to D converter (3) is connected to the processor means (4). The said 
processor means can be a "digital signal processor" (DSP) or a micro-controller 
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or an ASIC. A first input of the said processor means is connected to ROM (5) 
containing text to speech conversion and speech to text conversion software (5.1 
and 5.2 respectively). The one input-output of the said processor means is 
connected to RAM (6) which contains speech to text data (6.1, text to speech 
5 data 6.2 and commands and status data (6.3). A second input of the said 
processor means is connected to receiver (12) which receives data from the 
remote computer containing e-mail system. Another input-output of the said 
processor means is connected to the control panel containing switches and 
display (10). Another output of the processor means (4) connects to the input of 
10 D to A converter (7) which supphes analog signal to audio amplifier (8) which 
drives speaker (9) to produce audio output. 

In figure 4 the remote control unit (RCU) resides in the computer system and 
interacts with the OS through Remote Control Unit Driver (RCUD). The RCU 

15 receives signals from the remote control device RCD and interacts with the OS 
Kernel which gives instructions to appropriate device and other sub-systems 
including the speech software and the e-mail software. The OS Kernel 
coordinates tiie transfer of the data from the e-mail system to the remote control 
unit device driver (RCUD). The RCUD operates the remote control unit (RCU) 

20 hardware to transmit and receive data to/from the distant hand held remote 
control device. The OS software is built in such a way that the drivers are 
supported by means of appropriate libraries with support function calls to such 
device. 

25 The priority of the e-mail could be normal or urgent. An e-mail software could 
parse this information and provide the same to the user either by default or upon 
prompting. Another type of urgency information depends on e-mail contents. 
Here the user may be prompted for keywords or date and the e-mail software 
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can parse/search the e-mails for keywords and/or dates and provide appropriate 
information. 

Workiiig of the invention: 

5 User speech input is received by microphone (1), which converts the sound 
signals into electrical analog signals. These analog signals are amplified by 
microphone amplifier (2). The output fi^om the microphone amplifier is 
converted by A — D converter (3) into digital signals. These digital signals are 
processed by processor means (4) using speech to text converter programme 
10 (5.2) contained in ROM (5) and converted to speech to text data (6.1) and stored 
in RAM (6). This data is then transmitted by transmitter (11) to the remote 
computer containing the e-mail system (13). 

In the reverse process data transmitted by the e-mail system (13) is received by 
15 receiver (12) in the remote control device. The received data is processed by 
the processor means (4) using text to speech converter programme (5.1) 
contained in ROM (5) into text to speech data which is then stored in RAM (6). 
This data is further converted by D- A converter (7) into an analog signal which 
is applied to the input of audio amplifier (8) which drives speaker (9) to deliver 
20 output sound signals for the user. 

Control panel (10) containing switches and display devices is operated by the 
user for providing commands to the system. The processor means (4) receives 
the commands from the control panel (10) stores as commands and status data 
25 (6.3) in the RAM (6) and transmits to the remote computer system through 
transmitter (11). It also receives status information fi:^om the remote e-mail 
system (13) through receiver (12) and displays on the control panel (10) for the 
user's information. The messages received from the remote computer can also 
be displayed on the remote control device (RCD). The control panel (10) 
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typically includes switches for turning the remote control device on/ofif and 
buttons for incrementing or decrementing selected e-mail number, instructing 
for reading e-mail header information. It may further include buttons for 
instructing reading of e-mail content and number selection buttons for pointing 
5 to e-mail messages by number, starting accepting spoken instructions and 
buttons for interrupting a particular activity. 



In the computer containing the e-mail system, the data from the Remote Control 
Device (RCD) is received by Remote Control Unit (RCU) which is operated by 
10 the Remote Control Unit Device Driver (RCUD) software. The RCUD software 
then interacts with the OS Kernel (OSK) which transfers the received data to the 
e-mail system. In like manner, data sent by the e-mail system is transferred to 
the RCUD through the (OSK). RCUD then operates the RCU to transmit the 
data to the RCD. 

15 

The invention will now be described with reference to the following examples: 
Example 1. 

■ Receive e-mail 

■ Update e-mail count 

20 ■ Text to speech conversion 

■ Announce sender, subject, count of e-mail, urgency information 

■ Announce e-mail content if remote control device is used to instruct the 
same 

■ Detach attachments and announce text part if prompted 

25 

Example 2. 

■ User uses remote control device to point to latest unread e-mail 

■ The user can provide instructions to announce sender, subject, count of e- 
mail, urgency information 
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■ The user can use the remote control device to point to the next unread e- 
mail and request similar information. Buttons on the remote control 
device enable this. 



5 Example 3- 

■ User uses the remote control to point to the appropriate mail, subject 
wise, sender wise or count wise. 

■ The remote control prompts for the name of the sender or the text of the 
subject. 

10 ■ The user announces it aloud at the remote control device which has a 
facility to record voice. 

■ The remote control device transmits this information to the device in the 
computer system where speech to text conversion occurs and the 
respective e-mail message is accessed. 

15 ■ The user wants to send a reply to an e-mail and indicates so by using the 
remote control device. 

■ The remote control device prompts for sender, subject and content and 
attachment if any. 

■ The user provides the same and instructs to send the e-mail. 

20 

ADVANTAGES: 

This type of e-mail support is useful in the following scenarios: 

■ People with physical disabilities and who cannot move to or out of their 
25 work place very frequently. 

■ People who are blind can also use this system if they are trained to use 
the remote control device. 

■ This system will typically be most beneficial to reply/read short important 
messages. But is much better and more powerful than a cell phone or a 
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pager. That is because one can get all the power of e-mail whereas in the 
case of a cell phone one can only send a message to one person. 
While it may appear cumbersome or time consuming to expect a user to 
read e-mail content or attachment content, the facility is provided mainly 
to be used for short messages and for the sake of completion. 
In workplaces like construction sites, mines etc. users can do very well 
with just a hand held device and access e-mail and reply to the same. 
This invention can be used where there is no computer screen or monitor. 
This invention is different from the solutions provided based on WAP 
because it does not require Internet Service Provider (ISP) facility the 
remote control device is similar to the television remote control and its 
range has a distance limitation. 
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We claim: 



1. A computing system comprising a central processing unit, associated 
memory, input/output devices and containing an e-mail system, 
5 characterized in that it includes a remote control device comprising : 

means to convert speech to text. 

means to transmit the said text to the said e-mail system 
means to receive text from the said e-mail system 
means to convert the received text into speech 
10 - means to select and access the received e-mail in the said e-mail 

system. 



2. A computing system as claimed in claim 1 wherein said remote control 
device (RCD) further includes a means to display the received email 
1 5 messages. 



3. A computing system as claimed in claim 1 wherein the said remote 
control device is customized by the user to speak aloud the sender, date, 
subject, e-mail content and attachment (if required) for incoming mail. 

20 

4. A computing system as claimed in claim 1 wherein the said remote 
control device includes means for selecting email message or folder. 



5. A computing system as claimed in claim 1 wherein the said remote control 
25 device includes means for speaking aloud the summary of an e-mail folder. 



6. A computing system as claimed in claim 1 wherein the said remote control 
device is linked to the said computer through infra-red or ultrasonic or 
radio frequency waves. 
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7. 



A computing system as claimed in claim 1 wherein the said e-mail system 
provides a audio announcement whenever new messages are received. 



5 8. A computing system as claimed in claim 1 wherein the said remote control 
device comprising : 

a microphone connected to the input of microphone amphfier, 
the output of said microphone is connected to the input of an analog 
to digital converter (ADC). 
10 - the output of the said ADC is connected to the one input of a 

processor means, 

the second input of said processor is connected to a ROM (Read Only 
Memory) containing text-to-speech, conversion and speech to text 
conversion software, 

15 - a first input-output of said processor means is connected to Random 

Access Memory (RAM) which contains the speech-to-text data, text- 
to-speech-data, and command and status data, 

one output of said processor means is connected to a transmitter for 
transmitting the said data in RAM to a remote computing system 
20 containing an e-mail system, 

another input of the said processor means is connected to a receiver 
which receives the data from the remote computing system 
containing e-mail system, 

another input/output of the processor means is connected to a control 
25 panel containing switches and display elements, 

another output of the said processor means is connected to a digital to 
analogue convertor (DAC) which converts the data received after 
text-to- speech conversion into an analogue electrical signal and feeds 
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to the input of an audio amplifier which drives a loud speaker to 
generate audible sound. 

9. A computing system as claimed in claim 8 wherein the said processor 
5 means is a digital signal processor. 

10. A computing system as claimed in claim 8 wherein the said processor 
means is a micro controller. 

10 11. A computing system as claimed in claim 8 wherein the said processor 
means is an ASIC. 

12. A computing system as claimed in claim 11 wherein the said ASIC 
includes the entire circuit except the microphone, speaker, computer 

1 5 containing e-mail system and control panel. 

13. A computing system as claimed in claim 11 wherein the said ASIC 
contains ROM , RAM ^transmitter and receiver. 

20 14. A computer programme product residing on computer storage media 
containing computer readable devices embodied therein for causing a 
computer e-mail system to receive instructions and e-mail content from a 
remote control device and to send status and e-mail content to the said 
device. 
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15. A computer programme product as claimed in claim 14 wherein the said 
includes the facility for text-to-speech conversion. 
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16. In a computing system consisting of a central processing unit, associated 
memory, input/output devices and containing an e-mail system, a method 
for remotely controlling the said e-mail system using an audio support 
comprising: 

5 

sending e-mail by: 

receiving the speech input from the user, 
converting the speech input into the analog electrical signal, 
amplifying the said electrical signal and converting to digital form, 
10 - converting the said digitized to text, 

transmitting the said text to the computer containing e-mail system, 
receiving the transmitted text in the computer containing e-mail 
system, 

transferring the said text to the said e-mail system, 

15 

receiving e-mail by: 

transmission of data from computer containing e-mail system, 

receiving transmitted date at the remote control device, 
converting said received data from text to digitized speech, 
20 - converting the said digitized speech to analog form, 

amplifying the said analog signal and converting to speech for the 
user. 

17. A method as claimed in claim 16 wherein the email is selected through 
25 speech or through control panel. 

18. A method as claimed in claim 16 wherein the received mail message is 
displayed on the said remote control device (RCD) 



15 



ABSTRACT 



This invention relates to an apparatus, program product and a method for 
controlling an email system in a remote computer using a remote control device 
having an audio support, speech-to-text and text-to-speech facilities. 
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